Unified Image and Video Saliency Modeling

Visual saliency modeling for images and videos is treated as two independent tasks in recent computer vision literature. While image saliency modeling is a well-studied problem and progress on benchmarks like SALICON and MIT300 is slowing, video saliency models have shown rapid gains on the recent DHF1K benchmark. Here, we take a step back and ask: Can image and video saliency modeling be approached via a unified model, with mutual benefit? We identify different sources of domain shift between image and video saliency data and between different video saliency datasets as a key challenge for effective joint modelling. To address this we propose four novel domain adaptation techniques - Domain-Adaptive Priors, Domain-Adaptive Fusion, Domain-Adaptive Smoothing and Bypass-RNN - in addition to an improved formulation of learned Gaussian priors. We integrate these techniques into a simple and lightweight encoder-RNN-decoder-style network, UNISAL, and train it jointly with image and video saliency data. We evaluate our method on the video saliency datasets DHF1K, Hollywood-2 and UCF-Sports, and the image saliency datasets SALICON and MIT300. With one set of parameters, UNISAL achieves state-of-the-art performance on all video saliency datasets and is on par with the state-of-the-art for image saliency datasets, despite faster runtime and a 5 to 20-fold smaller model size compared to all competing deep methods. We provide retrospective analyses and ablation studies which confirm the importance of the domain shift modeling. The code is available at https://github.com/rdroste/unisalComment: Presented at the European Conference on Computer Vision (ECCV) 2020. R. Droste and J. Jiao contributed equally to this work. v3: Updated Fig. 5a) and added new MTI300 benchmark results to supp. materia

Assessing neural activity related to decision-making through flexible odds ratio curves and their derivatives

It is well established that neural activity is stochastically modulated over time. Therefore, direct comparisons across experimental conditions and determination of change points or maximum firing rates are not straightforward. This study sought to compare temporal firing probability curves that may vary across groups defined by different experimental conditions. Odds-ratio (OR) curves were used as a measure of comparison, and the main goal was to provide a global test to detect significant differences of such curves through the study of their derivatives. An algorithm is proposed that enables ORs based on generalized additive models, including factor-by-curve-type interactions to be flexibly estimated. Bootstrap methods were used to draw inferences from the derivatives curves, and binning techniques were applied to speed up computation in the estimation and testing processes. A simulation study was conducted to assess the validity of these bootstrap-based tests. This methodology was applied to study premotor ventral cortex neural activity associated with decision-making. The proposed statistical procedures proved very useful in revealing the neural activity correlates of decision-making in a visual discrimination task.status: publishe

A role for the ventral premotor cortex beyond performance monitoring

Depending on the circumstances, decision making requires either comparing current sensory information with that showed recently or with that recovered from long-term memory (LTM). In both cases, to learn from past decisions and adapt future ones, memories and outcomes have to be available after the report of a decision. The ventral premotor cortex (PMv) is a good candidate for integrating memory traces and outcomes because it is involved in working-memory, decision-making, and encoding the outcomes. To test this hypothesis we recorded the extracellular unit activity while monkeys performed 2 variants of a visual discrimination task. In one task, the decision was based on the comparison of the orientation of a current stimulus with that of another stimulus recently shown. In the other task, the monkeys had to compare the current orientation of the stimulus with the correct one retrieved from LTM. Here, we report that when the task required retrieval of the stimulus and its use in the following trials, the neurons continue encoding this internal representation together with the outcomes after the monkey has emitted the motor response. However, this codification did not occur when the stimulus was shown recently and updated every trial. These results suggest that the PMv activity represents the information needed to evaluate the consequences of a decision. We interpret these results as evidence that the PMv plays a role in evaluating the outcomes that can serve to learn and thus adapt future decision to environmental demands